Two notes on this post:
- This post discusses flaws in a particular published cost-effectiveness estimate for deworming. It should not be taken as a general argument against deworming as a promising intervention, and it does not address various other publications on deworming including the 2003 paper by Edward Miguel and Michael Kremer.
- Prior to publication, we sent a draft of this post to several relevant scholars including the authors of the estimate. They have reviewed our work and confirmed the major errors we point out.
Over the past few months, GiveWell has undertaken an in-depth investigation of the cost-effectiveness of deworming, a treatment for parasitic worms that are very common in some parts of the developing world. While our investigation is ongoing, we now believe that one of the key cost-effectiveness estimates for deworming is flawed, and contains several errors that overstate the cost-effectiveness of deworming by a factor of about 100. This finding has implications not just for deworming, but for cost-effectiveness analysis in general: we are now rethinking how we use published cost-effectiveness estimates for which the full calculations and methods are not public.
The cost-effectiveness estimate in question comes from the Disease Control Priorities in Developing Countries (DCP2), a major report funded by the Gates Foundation. This report provides an estimate of $3.41 per disability-adjusted life-year (DALY) for the cost-effectiveness of soil-transmitted-helminth (STH) treatment, implying that STH treatment is one of the most cost-effective interventions for global health. In investigating this figure, we have corresponded, over a period of months, with six scholars who had been directly or indirectly involved in the production of the estimate. Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. We came to this conclusion a year after learning that the DCP2’s published cost-effectiveness estimate for schistosomiasis treatment – another kind of deworming – contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY. (This figure appears, correctly, on page 46 of the DCP2.)
We do believe that the corrected DCP2 calculations are too harsh on deworming; our best estimate of the cost-effectiveness of deworming is in between the corrected and uncorrected DCP2 figures, at $30-$80 per DALY. In addition, there are strong arguments for deworming as an excellent intervention that do not depend on these figures. Overall we consider deworming a highly promising (though not the single most promising) intervention; we will be discussing our thoughts on this intervention further in the future. This post focuses not on deworming in general, but on the DCP2 figures and what lessons we should take from the flaws in them.
- The estimates on deworming are the only DCP2 figures we’ve gotten enough information on to examine in-depth. Getting to this point took a lot of work and communication with a number of different scholars, so we aren’t sure of the extent to which other estimates might also turn out to be flawed if examined closely.
- We believe that the errors we’ve found in the estimate would have been caught by a helminth expert independently examining the estimate. Therefore, the presence of these errors implies to us that there has been no such examination. If this is the case, it would argue against the reliability of the DCP2’s estimates in general.
- We’ve previously argued for a limited role for cost-effectiveness estimates; we now think that the appropriate role may be even more limited, at least for opaque estimates (e.g., estimates published without the details necessary for others to independently examine them) like the DCP2’s.
- More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust – no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the world’s foremost experts on soil-transmitted helminths.
- It is possible that we have made errors in our corrections to the calculation. One of the reasons we go to great lengths to be transparent is because we want our errors to be caught as quickly as possible.
Outline for the remainder of this post:
- About the DCP2’s estimate
- Why we decided to look into the DCP2’s deworming estimates
- Our process for investigating the estimate
- Apparent errors in the spreadsheet used to calculate the cost-effectiveness of STH treatment
- Our independent estimate of the cost-effectiveness of STH treatment
- Implications for donors interested in deworming
- Our takeaways
About the DCP2’s estimate
The DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation.
The Gates Foundation also appears to have invested substantially in the dissemination of the DCP2’s findings, including a $4.4 million grant to the Population Reference Bureau to “disseminate key messages from [the DCP2].”
The DCP2 aims to estimate the cost-effectiveness of different health interventions, in terms of dollars per disability-adjusted life-year (DALY) saved, in order to prioritize the most cost-effective interventions–the ones that will have the largest effects in reducing mortality and morbidity for a given amount of funding. The DCP2’s published estimates imply that soil-transmitted helminth (STH) treatment is one of the cheapest ways to improve health: the same “amount of health” could be provided by spending $1 on STH deworming or roughly $34 on family planning programs or more than $90 on treating drug-resistant tuberculosis. In fact, it appears that the DCP2 rates STH treatment as the second most cost-effective health intervention of all, behind only hygiene promotion (p. 41).
The DCP2’s cost-effectiveness estimates for deworming have been cited widely to advocate a greater focus on treating STH infections, including in:
- an article (PDF) in The Lancet
- a report (PDF) by REACH, a consortium of large international NGOs and other organizations working to end child hunger, which labeled deworming one of 11 “promoted interventions”
- the most-cited paper (PDF) published in the journal International Health
- an editorial by Peter Hotez, a co-founder of the Global Network for Neglected Tropical Diseases, which has received more than $40 million in funding from the Gates Foundation
- work by charity evaluators, such as GiveWell, Giving What We Can, and the University of Pennsylvania’s Center for High Impact Philanthropy.
Why we decided to look into the DCP2’s deworming estimates
We undertook this research because:
- We wanted to do a case study of a cost-effectiveness estimate from the DCP2, understanding the full details of what goes into it and where the room for error is.
- We were particularly curious about the estimate for treatment of soil-transmitted helminths since the published $3.41 per DALY averted figure didn’t seem to sync with what we knew about the costs and effectiveness of STH treatment (or the independent estimate of $280/DALY given by another study, as we’ve mentioned previously).
- We also wanted to focus on STH treatment since the DCP2 rates it as the second most cost-effective health intervention of all, behind only hygiene promotion.
- Finally, we wanted to learn more about deworming after Elie visited the Schistosomiasis Control Initiative in London and we became more optimistic about this organization than we had been.
Our process for investigating the estimate
GiveWell took the following steps to investigate the DCP2’s estimate for the cost effectiveness of STH deworming:
- We initially contacted Peter Hotez, the lead author of the DCP2 chapter on intestinal nematode infections; he sent us several papers on the costs and effectiveness of deworming and referred us to another scholar to explain the calculation that the DCP2 had published.
- This scholar, in turn, referred us to two more, who sent us further references in response to our questions.
- At this point we had an extended back-and-forth trying to understand the details of the calculation that had been done, and since we weren’t sure we would reach a conclusion on this, we asked volunteer Jonah Sinick to use all the references we’d been sent to create his own best guess estimate for the cost-effectiveness estimate of deworming. This estimate implied a significantly higher cost per DALY than the published figure, which seemed strange since we were now using the references and inputs suggested to us by the chapter authors.
- The scholars we had been corresponding with sent us a spreadsheet with the full details of the calculation, as well as an accompanying table, which we will call Table 9, that had been used to input some of the figures in the spreadsheet. Here is the PDF of Table 9 that we were sent.
- However, the interpretation of the numbers from Table 9 was still unclear to us. Table 9 is not clearly labeled; the scholars involved in the calculation appeared to have conflicting interpretations of what the numbers meant, and both meanings were highly counterintuitive to us (details below).
- So we contacted another scholar who had worked on Table 9 to get her help in interpreting it. She sent us the full paper from which Table 9 was taken, Intestinal Nematode Infections, and this paper appeared to have a different interpretation of Table 9 than the spreadsheet’s. We confirmed this with her.
- We also found the disability weights being used counterintuitive, and after some investigation we received confirmation that they were erroneous (details below).
- All in all, we found five errors in the estimate, not all of which were attributable to the creator of the spreadsheet.
Problems with the official estimate of the cost-effectiveness of deworming
The basic approach of the estimate is to:
- Calculate the benefits of deworming by
- Starting from a population of schoolchildren being dewormed;
- Estimating the percentage of these children suffering from different symptoms of infection;
- Using the above, estimate the number of children cured of these symptoms (the estimate assumes that they are cured for exactly one year, since reinfection can occur after deworming)
- Incorporating the severity of symptoms to arrive at DALYs saved by the deworming
- Separately calculate the costs of deworming this population of schoolchildren, and divide costs by DALYs to obtain the cost per DALY.
When we examined the details of the official estimate, it struck us that nearly all of the DALYs saved (i.e., nearly all of the benefit) were coming from the reduction of a single symptom of a single worm infection: cognitive impairment due to ascariasis (we abbreviate this as CIDTA). Specifically, the figures going into the estimate implied that:
- In a hypothetical population of 208,530 children (age 5-14 in Latin America) treated, 45,060 suffer from CIDTA. (Cells C44 and L44 in “ascariasis” sheet). That’s about 22%.
- The disability weight of CIDTA is 0.463 (cell E8). While these figures are difficult to interpret, this implies that having CIDTA is about half as bad as being dead (disability weight 1.0), and only slightly less debilitating than being blind (disability weight 0.6). (See the official list of disability weights published alongside the DCP2.) These figures implied (to us) that CIDTA was not a matter of subtle cognitive impairment, but of mental handicap so severe as to truly prevent normal functioning.
- The intervention in question – a single dose of albendazole – could completely restore normal mental functioning (i.e., completely eliminate disability associated with CIDTA) for one year.
These implications didn’t sync with the information we had from other sources, such as the Global Burden of Disease (GBD) report published alongside the DCP2.
- If ascariasis caused this sort of symptom, we’d expect to see much more focus on ascariasis (relative to other helminth infections) in the global health and deworming communities.
- In addition (as we observed when trying to reconcile the official estimate with our own estimate), if 22% of the 110 million 5-14 year olds in Latin America (GBD, 198-199) had a disability with weight 0.463, then this – alone – would result in 11.2 million DALYs lost to ascariasis per year in this region (22% * 110 million * 0.463). However, the official DALY burden for this ascariasis (all symptoms) among this population is only 31,000 (GBD, 198-199) – in fact, the worldwide DALY burden for ascariasis is only 915,000 (GBD, 180-181).
We therefore did further investigation on the CIDTA symptom – both how prevalent it is and how severe it is. It turns out that the official calculation significantly overstates both. For example, among 5-14 year olds in Latin America, CIDTA affects about 0.23% of the population – not 22.6% as the official calculation suggests – and its correct disability weight is 0.024 (the same severity as anemia), not 0.463.
Specifics of these errors:
- Prevalence of CIDTA. The official calculation starts from a hypothetical population of 1 million people of all ages, then calculates the number of 5-14 year olds (per million people) using demographic data, then takes the number of CIDTA cases directly from Table 9 (this figure is multiplied by 10 before being put in the official spreadsheet). For example, for 5-14 year olds in Latin America, Table 9’s “A/B” column has the figure, “4506”; the official calculation records “45060” for the number of CIDTA cases among 5-14 year olds.
The labeling of Table 9 is ambiguous and doesn’t make it clear whether this is the intended meaning of the figures. We contacted one of the original authors who wrote the paper from which Table 9 is taken, received a copy of the (unpublished) paper from her, discussed it with her, and found that this figure’s intended interpretation is different from the official calculations, in two ways:
- The figure in the “A/B” column refers number of people at risk for a given symptom, not the number of people suffering from that symptom. These are equivalent for Type A and Type C symptoms, but not for Type B symptoms including CIDTA. Intestinal Nematode Infections (PDF), the working paper that contains Table 9, says that “in any annual cohort of heavily infected children some 5% suffer [Type B symptoms, which are the only symptoms that have life-long effects]” (p. 26). Using the figures as the official calculation did would therefore lead to a 20x overstatement in the prevalence of CIDTA.
This mistake applies not just to cognitive impairment due to ascariasis, but also to cognitive impairment due to trichuriasis and hookworms, similarly leading to a 20x overstatement of the prevalence of cognitive impairment due to those infections as well.
-
The figures in Table 9 refer to the number of children at risk, per 100,000 children of the age group indicated in the row. For 5-14 year olds in Latin America, the figure (for symptoms “A/B”) is “4506”; this means that 4506 out of 100,000 5-14 year olds are at risk for CIDTA. This in turn means that 45060 of every million 5-14 year olds are at risk. However, the official calculation assumes 45060 cases not for one million 5-14 year olds, but for only 208,530 5-14 year olds (which is the number of 5-14 year olds one would expect in a population of 1 million people across the three age groups). Thus, this difference results in overstating the prevalence of CIDTA by about 5x.
This mistake applies to each of the symptoms of all three soil-transmitted helminths, not just to CIDTA, and therefore leads to an overstate of the prevalence of every symptom of STHs by about 5x.
Bottom line – the correct interpretation of Table 9 (for 5-14 year olds in Latin America) is that 45060 out of every million 5-14 year olds are at risk for CIDTA, and 5% of these actually have it – so 2253 out of every million 5-14 year olds have CIDTA. The official calculation assumes that in a population of 208,530 5-14 year olds, 45060 have CIDTA. The same types of errors apply to the other regions and conditions as well.
- The figure in the “A/B” column refers number of people at risk for a given symptom, not the number of people suffering from that symptom. These are equivalent for Type A and Type C symptoms, but not for Type B symptoms including CIDTA. Intestinal Nematode Infections (PDF), the working paper that contains Table 9, says that “in any annual cohort of heavily infected children some 5% suffer [Type B symptoms, which are the only symptoms that have life-long effects]” (p. 26). Using the figures as the official calculation did would therefore lead to a 20x overstatement in the prevalence of CIDTA.
- Severity of CIDTA. The disability weight of 0.463 is correctly transcribed from the Global Burden of Disease official disability weights, which in turn takes the figure from the earlier 1996 edition (which we examined in a library). However, we still found this figure odd because of the contrast with the other two kinds of helminth infections:
Helminth type Symptom A – disability weight Symptom A – description Symptom B – disability weight Symptom B – description Symptom C – disability weight Symptom C – description Ascariasis 0.006 Reduction in cognitive ability in school-age children, which occurs only while infection persists 0.463 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.024 Blockage of the intestines due to worm mass Trichuriasis 0.006 Reduction in cognitive ability in school-age children, which occurs only while infection persists 0.024 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.114-0.138 Rectal prolapse and/or tenesmus and/or bloody mucoid stools due to carpeting of intestinal mucosa by worms Hookworm NA NA 0.024 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.024 Anemia due to hookworm infection It looked to us as though the weights may have been switched, in the case of ascariasis, for symptoms B and C. We contacted Colin Mathers, the second-listed author on the Global Burden of Disease publication, and he confirmed to us that the weights are in fact switched, stating, “We also noticed this and corrected it in the spreadsheets for WHO estimates, but possibly it has remained uncorrected in some of the summary tables of weights.” Thus, CIDTA’s correct disability weight is 0.024, but the published disability weight in both editions of the GBD – and the weight used in the official cost-effectiveness calculation – is 0.463.
We created a version of the official calculation that corrected for the above errors, as well as two other errors that we found in the process of checking the calculation as thoroughly as we could. (See Footnote 1 below.) Our version is here (XLS).
This calculation leads to a revised cost-effectiveness estimate of $326.43 per DALY, rather than the $3.41 per DALY in the original.
The DCP cost-effectiveness estimates only took into account short term effects of the three diseases, even though they have some long term effects. This seems to have been an intentional decision rather than an error, but our feeling is that a best estimate of the true cost-effectiveness of deworming would likely take these long-term effects into account. We therefore created another version of the estimate that does so, as best as we can. (See Footnote 2 below.) Taking these long-term effects into account, our cost-effectiveness estimate for STH treatment moves to $138.28 per DALY.
These corrections also have implications for the cost-effectiveness estimate for combination deworming (simultaneously addressing both STH and schistosomiasis, another type of infection). The DCP2 reports a cost-effectiveness estimate of $8-$19/DALY averted for combined treatment, depending on whether generic or brand-name drugs are used for schistosomiasis treatment. Using our overall best guess for the revised DCP2 estimate for STH of $138.28/DALY and the DCP2’s estimate for generic schistosomiasis drugs of $336/DALY (note that this is incorrectly presented as “$3.36/DALY” on page 476, but the correct figure – without the erroneous decimal point – appears on page 46), we estimate the cost-effectiveness of a combined program, according to the DCP2, as $177/DALY. Ignoring the long-term effects of STH treatment, as the DCP2 does, changes that figure to $272/DALY.
In our first email to the author of the spreadsheet, we had only caught the first four of the five errors mentioned above, and made substantial mistakes in our attempts to take long-term effects into account. It was only when we checked the figures later that we noticed both of these mistakes. Mistakes are easy to make in this type of situation (for an interesting study on spreadsheet mistakes, see here). Transparency is the best way we can think of to avoid such mistakes. Now that we’ve published the spreadsheets, we look forward to hearing about any other mistakes you find – in the original or ours.
Our independent estimate of the cost-effectiveness of STH treatment
At the same time we were working through the DCP cost-effectiveness estimate for STH deworming, Jonah Sinick, a GiveWell volunteer, was working on an independent set of cost-effectiveness estimates for deworming, separately for both STH and a second type of worm-based disease, schistosomiasis. His report on the results is now available here. His bottom-line best guess for the cost-effectiveness of STH deworming is $82.54/DALY. Jonah’s calculation implicitly takes long-term effects into account, as we do in our more optimistic version of the calculation (the one that comes to $138.28 per DALY). Most of the discrepancy between Jonah’s $82.54/DALY figure and our $138.28 figure can be explained by the DCP’s use of a much higher cost-per-child treated ($0.225 vs. $0.085), though Jonah also finds different levels of disease burden and treatment effectiveness. (See footnote 3 below.)
Jonah also found more promising results for schistosomiasis treatment, another form of deworming that (as mentioned above) can be combined with STH treatment. His estimate ranges from $28.19-$70.48/DALY for schistosomiasis deworming. This is much more optimistic than the DCP’s estimate of $336-$692/DALY because Jonah finds, following the current consensus in the literature, a much higher disability weight for schistosomiasis than the DCP used (0.02-0.05 vs. 0.005-0.006). The DCP’s higher cost-effectiveness estimate also assumes using much more expensive brand-name drugs, while the lower estimate, like Jonah’s, assumes generics.
Conservatively combining Jonah’s estimates for the cost-effectiveness of schistosomiasis and STH deworming (by assuming that no delivery costs are saved), we reach an estimate of $32-72/DALY, depending on the disability weight of schistosomiasis. More liberally assuming that a combined program would eliminate delivery costs equal to half the per-child cost of STH treatment, Jonah’s estimate of the cost-effectiveness of a combined program ranges from $29/DALY to $66/DALY, depending on the disability weight of schistosomiasis.
Implications for donors interested in deworming
These estimates are only a small part of the picture, in our view, regarding how promising deworming is as an intervention. We will be writing more about this in the future.
However, we think it is important to note that the DCP2’s original published figures implied that deworming is among the most cost-effective interventions listed in the publication; with errors corrected, it appears comparable to treating drug-resistant tuberculosis; taking into account long-term effects, it seems comparable to providing family planning services. Neither of those interventions are traditionally considered especially cost-effective. (Note that that according to the DCP2’s original estimate, STH deworming is 30-100X more cost-effective than those interventions.)
Whether or not the long-term effects are taken into account, the corrected DCP2 estimate of STH treatment falls outside of the $100/DALY range that the World Bank initially labeled as highly cost-effective (see page 36 of the DCP2.) With the corrections, a variety of interventions, including vaccinations and insecticide-treated bednets, become substantially more cost-effective than deworming.
The more important takeaway, for us, concerns the DCP2’s cost-effectiveness estimates in general. We believe that the errors we’ve found in the estimate – described above – would have been caught by a helminth expert independently examining the estimate. Therefore, the presence of these errors implies to us that there has been no such examination. If this is the case, it would argue against the reliability of the DCP2’s estimates in general. We have not done similar investigations of other DCP2 estimates, and given the process it took to get the details of this one, we are not planning to do many more until and unless the details of estimates become available publicly.
Our takeaways
- We’re now much more hesitant to place any weight on DCP2 cost-effectiveness figures except where we can fully understand and check the calculations.
- More generally, we feel this case illustrates how opaque, formal calculations can obscure important information and demonstrate high sensitivity to minor errors. We see this as support for our position that formalized cost-effectiveness analysis can do more harm than good in trying to maximize actual cost-effectiveness.
- Explicit cost-effectiveness estimates will continue to play a relatively small role in our decisions between top charities, though we will still use them in deciding which charities are potential top candidates.
- We’re continuing to investigate deworming as a promising intervention, but one of the most encouraging figures widely cited in its favor appears deeply flawed.
- Transparency is crucial. Had the scholars we discussed these issues with been less willing to engage with us, or had we been unable to find Intestinal Nematode Infections or the spreadsheet, these substantial errors would not have come to light.
Footnote 1: The other two problems we found in the calculation both have to do with the burden of trichuriasis:
- The spreadsheet swaps the disability weights for Type B and C symptoms of trichuriasis. In the Global Burden of Disease and Risk Factors (GBD) 1990, which the spreadsheet cites, the Type B symptom of trichuriasis is cognitive impairment, which has a disability weight of 0.024, while the Type C symptom is massive dysentery syndrome, with disability weights ranging from 0.116 to 0.138. In the ‘trichuriasis’ sheet of the spreadsheet, Type B morbidity has disability weights ranging from 0.116 to 0.138 while Type C morbidity has the lower disability weight of 0.024. In the original calculation, this leads to an overestimate of the burden of trichuriasis by nearly 4x, but once the main errors described above are corrected, correcting this error actually makes STH treatment appear more cost-effective.
- The spreadsheet uses a duration of .05 years for trichuriasis symptom Type C, while Intestinal Nematode Infections suggests that the duration for trichuriasis symptom Type C should be 12 months (pg. 24). This mistake likely occurred because the duration for ascariasis symptom Type C is .05 years.
In the corrected spreadsheet, sheets ‘a.3’, ‘t.5’, and ‘h.3’ contain our corrections to all five of the issues we have identified (for ascariasis, trichuriasis, and hookworm respectively). Most of the corrections should be fairly self-explanatory, but please don’t hesitate to email us or comment here if you have questions. We corrected the second main error above by changing the population of 5-14 year olds treated to 1,000,000 (see, e.g., sheet ‘a.3’ cell C23).
Footnote 2: The Type B symptom of all three diseases treated by STH deworming is called “cognitive impairment,” has a disability weight of 0.024, and lasts a lifetime once it develops. Intestinal Nematode Infections implies that 3% of the population at risk for symptom B (that is, 3% of the population listed in the A/B columns in Table 9) newly acquires a lifelong disability each year (pg. 26). We therefore altered the calculation to reflect lifelong (not just 1-year) benefits for these 3% (replacing the 5% listed in #2 above because that 5% is the total proportion infected during a given year, not the total proportion newly infected). At the same time, we also changed DALYs saved due to prevented mortality to compound to the end of life, rather than just counting the one year of life saved during the treatment. (This, arguably, is an actual error in the DCP2 process, not just a disagreement about how to take long term effects into account. When an intervention prevents someone from dying, it does not seem reasonable to count just one extra year of life saved.)
Footnote 3: We also looked into the possibility that the disability weights for helminth infections are “too low,” as implied by a passage in the DCP2:
The Disease Control Priorities Project helminth working group has determined that the WHO global burden of disease estimates are low because they do not incorporate the full clinical spectrum of helminth-associated morbidity and chronic disability, including anemia, chronic pain, diarrhea, exercise intolerance, and undernutrition (King, Dickman, and Tisch 2005). (DCP2, pg. 471)
Based on our review of the literature and correspondence with relevant scholars, we believe this argument has never been raised specifically in respect to STHs; most of the papers about it are about schistosomiasis, another type of worm infection. There is one paper (Chan 1997) that appears to imply a higher disability burden for STHs than the standard burden, which gives rise to Jonah’s more optimistic STH cost-effectiveness estimate of $11.25/DALY. We think the data from that paper is no longer credible: it appears to have been based on a lower worm threshold for experiencing morbidity than further research has found appropriate (Brooker 2010). Furthermore, the cited source of the relevant data is a working paper, the published version of which does not contain the data cited.
Comments
Very interesting reading, thanks. It’s underappreciated, I think, that spreadsheets are just a form of programming language* – and we all know how reliable and bug-free programs tend to be…
* Haskellers quip that Excel is the world’s most popular functional language (and also its most crippled, not having even first-order functions)
Thanks for producing this excellent and important research.
This was a significant update for me toward taking the DCP2 calculations less seriously. At the same time, I fear the conclusion in this post goes too far. I definitely would not be “hesitant to place any weight on DCP2 cost-effectiveness figures except where we can fully understand and check the calculations.” I imagine you don’t mean this literally. (You would not, I take it, hesitate between funding a charity with an intervention sampled from the bottom of the DCP2 rankings and funding a charity with an intervention sampled from the middle of the DCP2 rankings, given no other information.)
Some reasons not to go so far:
(i) This is only one DCP2 estimate.
(ii) It is an estimate of an intervention whose cost-effectiveness was more poorly understood than the cost-effectiveness of some of the other highly-rated health interventions. (I have limited information about claim (ii), but Holden expressed it to me in conversation recently. I also heard Chris Murray express some skepticism about the deworming estimates.)
In addition, this report mentions that Jonah’s estimate implicitly accounted for the long-term effects of deworming. But I see no term for this in Jonah’s equations. (Moreover, in a previous conversation with Jonah, I mentioned to him that his model did not take account of these effects, and he agreed that it did not.) Are the long-term effects implicit in Jonah’s calculation in some non-obvious way, was that claim a mistake, or am I somehow confused about this?
On further reflection, point (ii) is less important than I made it sound. It was not previously suggested that the DCP2 authors were making a significant number of computational errors and mix-ups that would cause the estimate to be off by a factor of 100, and the fact that deworming was more poorly understood (if true) would not have made this kind of error much more likely in the case of deworming than any other intervention.
However, such an error could still be somewhat more likely. After all, if the cost-effectiveness of some intervention were antecedently well-understood, one would have a better idea of what to expect regarding the intervention’s cost-effectiveness, and be more likely to detect an error that was off by a factor of 100x. In that case, the huge difference between one’s estimate and previous estimates would be grounds for checking the estimate again.
Hi Nick,
Thanks for the comments.
A few points in reply:
As I point out above, these errors would almost certainly have been caught by a helminth expert double checking the calculation. The DCP2’s failure to ensure that happened seriously hampers my assessment of its credibility.
Bottom line: we have many sources of information about charities. Given the proven possibility for large errors in opaque estimates like the DCP2’s, we don’t see much reason to rely on them in the future.
This is great stuff. I’m very impressed, and think that this research is extremely valuable. If you had asked me two years ago about how likely it is that DCP2 would have the sorts of presentation errors that we’ve found (like the decimal point), I would have said it’s unlikely; if you’d asked me about these calculation errors, I would have said ‘very unlikely’. I think there are pretty reasonable grounds for not treating the DCP2 as the authoritative source on cost-effectiveness that some of us (including me) had thought it to be.
Having said that, I just want to make some clarifications, so that we draw the right conclusions from the above.
In particular, we should distinguish three different questions:
1. How heavily should we rely on the stated cost effectiveness from the DCP2, as listed in the table of interventions?
As you’ve shown, I think we should be wary of relying solely on them. DCP2, of course, still provides evidence, but the value of checking that evidence against other estimates is very high indeed.
2. How heavily should we rely on cost-effectiveness estimates in general?
The main thing I’m worried about is inferring, from the fact that we should decrease reliance on DCP2, that we should decrease reliance on cost-effectiveness estimates in general. What we ultimately care about is how much benefit we can give people per $ invested in a charity. We shouldn’t lose sight of that.
In particular, your research shows, in terms of the global allocation of funds, that there should be *more* funding of cost-effectiveness research; because there are such great and easy gains to be had by simple improvements of the estimates. Your research also shows, in terms of what charity evaluators should be doing, that we need to spend *more* time looking into the literature of economic estimates of cost-effectiveness, because we don’t have one authoritative source.
In general, we should be wary of inferring from the fact the one source of X is less trustworthy than we thought to the conclusion that X, in general, isn’t to be trusted.
3. Should we think that deworming charities are the expectedly best charities to fund?
The above research should, I think, make one update in favour of ‘no’ to this question.
But it should be borne in mind that, somewhat counterintuitively, it’s not always true that gaining new evidence that disconfirms a proposition should lower one’s credence in that claim. If one is setting out to look for disconfirming evidence for a proposition, and one is not also looking for confirming evidence, and one then finds less disconfirming evidence than one would expect, then one should update in favour of that proposition.
This is relevant because I get the sense that GiveWell takes a deliberately critical stance when evaluating any intervention type or charity. So, at the end of GiveWell’s investigations, we need to ask not “How much disconfirming evidence is there?” but rather “How much disconfirming evidence is there, relative to what I would have expected prior to investigation?”
Even this question doesn’t give us quite what we want. Because it could just be that the noise of the data is greater than we would have expected (i.e. charity evaluation is harder than we thought!). This would mean that one would find a greater amount of disconfirming evidence, relative to what one would have expected prior to investigation, even when the expected cost-effectiveness is just the same.
I think that this could explain why, after GiveWell’s corrections, the DCP2 estimates for STHs and Schisto are so low. GiveWell’s research shows that the DCP2 estimates are extremely noisy. If we were to look for confirming evidence, then perhaps we’d find that severe errors (currently undiscovered) in the DCP2 calculation means that, by their own lights, they should have given a much more favourable estimate of STH’s cost-effectiveness than GiveWell’s corrected estimate.
Having said that, I think that the errors discovered by GiveWell do provide a greater amount of disconfirming evidence than I would have expected prior to investigation; and that this isn’t just attributable to noise. But it’s worth bearing in mind that we should also be looking for evidence in favour of higher cost-effectiveness of deworming as well. This could significantly increase our credence in deworming’s high cost-effectiveness.
To illustrate:
Reasons why deworming might be even better than the DCP2 estimates make out (this is without deliberately looking for good things; and there might be more I’ve currently forgotten about, as these are off the top of my head):
Fewer treatments per child (i.e. 2-3) in long run
Economic and educational benefits of deworming.
Effect on other illnesses. E.g. HIV/AIDS. Not accounted for in contemporary cost-effectiveness studies.
Extra economic benefits of improving quality of life rather than extending lives
Because of the way that DALYs are calculated, Years of Life Lost are are inflated in importance relative to Years Lived with Disability.
Reasons why specific deworming charities like SCI might be even more cost-effective than cost-effectiveness of STH treatment in general (again off the top of my head):
SCI administers a combination package of drugs, to treat the 7 most prevalent NTDs. The drug costs are very low or nonexistent, and so I’d expect the combined cost-effectiveness to be much higher than the cost-effectiveness of one intervention on its own.
There are some elimination programs (e.g. Zanzibar)
Benefits from government handover
Leverage of governmental resources (which would have been spent on something less cost-effective).
This isn’t the place to get into the nitty gritty of deworming charities; but the important lesson is that we should be impartial in our search for confirming and disconfirming evidence. If we don’t do this, then we shouldn’t automatically decrease our credence in a proposition given that new disconfirming evidence comes to light.
Like I say, this is extremely important research that you’ve done, and it should make us more wary of relying on the DCP2.
Hi Alexander,
As long as we agree that it would be irrational to give no weight to DCP2 estimates, that’s all I wanted to establish with the hypothetical example.
When I said it was one estimate, I suppose I should have said that it was one DCP2 chapter, all done by the same group. Different groups may have been run significantly differently, so it is not transparent how much to downgrade other estimates by the DCP2.
I’m not sure what to make of your point that the chapter errors are not indicative of “a process that ensures correct results” or the claim about who has the “burden of proof.” I don’t think you’d have found me, or anyone else, really, ready to sign up for the view that the DCP2 process “ensures correct results”, though your research has significantly decreased my confidence in this process. However, I don’t think that treating some estimate as decision-relevant requires thinking that the estimate was produced by a perfectly or extremely reliable process. I just do my best to assign a subjective probability to these kinds of things and act on the basis of that assignment. I currently have a 90% subjective probability that if someone looked at another DCP2 estimate for a highly-rated intervention, we would not find that the estimate was off by a factor of 100 due to dimple computational errors, and a 70% subjective probability that we would not find it to be off by a factor of 10 due to simple computational errors. (Previously, I might have had much higher probabilities in these claims, like 97% and 90%.) For me, this is enough to make the use of such estimates an important tool for analysis. I would be curious what your subjective probabilities are for these claims, since I might update toward your position.
One important decision-making upshot of this research for me is that before I make recommendations on the basis of cost-effectiveness research like this, I will make every reasonable attempt to obtain the original calculations or reproduce them myself. However, if I cannot do this, I will not act as if the estimate had negligible weight.
Nick,
I find your comment above abstract to the point that I don’t know what you have in mind; are there particular interventions that you think that people interested in efficient philanthropy should focus more on or less on based on the DCP2 cost-effectiveness estimates?
Will:
Thanks for the comments. A few responses:
As Holden and I have been saying, we agree with the goal of maximizing expected value. It is becoming less clear whether explicit cost-effectiveness estimates, even “good” ones, are a good way to maximize expected value. I don’t think anyone is losing sight of the goal of doing as much good as possible with money. But I also don’t think that, say, using the Copenhagen Consensus or WHO-Choice is the best strategy for getting better estimates. We’ll be discussing this more in future posts.
As I mentioned above, we had a number of reasons for looking into the DCP2 estimate; trying to debunk the DCP2 (or deworming) wasn’t one of them. Like you, we started off thinking that the DCP2 was a relatively credible source of cost-effectiveness estimates. In a 2008 blog post, for instance, we said that we were trying to get in touch with the DCP2 authors and lauded them for their thoroughness, while expressing frustration that the inputs to the calculation weren’t public. We’re also seriously considering recommending SCI, so understanding deworming is very important to us. We were very surprised that there were these mistakes in the DCP2 cost-effectiveness estimate.
I want to note that the mistakes we found point both directions. If you look at our corrected spreadsheet, you’ll see that the estimated cost-effectiveness when just errors 1-3 are fixed is $515/DALY. Once the two trichuriasis errors are fixed, it drops to $326.43, and once long term effects are taken into account, it goes to $138. There were numerous errors pointing in both directions, but the errors that overestimated the cost-effectiveness were larger.
In addition, we think Jonah’s calculation, which is much more optimistic about deworming, seems superior to the DCP2’s.
As to the points you raise in favor of deworming, we have written about many of these in the past (example). Many apply to the landscape of cost-effectiveness estimates in general. Our blog post is not intended as a summary of all possible issues with the DCP2 calc, but as an explanation of a few previously unknown issues.
When you say that “The above research should, I think, make one update in favour of ‘no’ to [the question, “should we think that deworming charities are the expectedly best charities to fund?”],” are you saying that you now think the answer is no, or just that there is less evidence for a yes answer?
We agree that the (cost-)effectiveness of particular deworming charities is up in the air right now. We’re continuing our research in this area.
Nick:
A couple more points:
We don’t know how the other cost-effectiveness estimates were made; they may have been better. As far as we can tell, the people who did the deworming cost-effectiveness estimate were also the ones who wrote the cost-effectiveness chapter in the DCP2 and were involved in many of the other chapters.
As we’ve discussed with GWWC elsewhere, the DCP2 estimates appear to be approximately log-normally distributed. More than 97% of the DCP2 estimates fall within two orders of magnitude of the mean, and 69% fall within one order of magnitude (a factor of 10) of the mean. I’ve attached the spreadsheet we used for this calculation here if you’re curious. I think we’ve sent a variant of it to you before.
Thus, according to your subjective probabilities, one can chalk the entire variance in DCP2 cost-effectiveness estimates up to “simple computational errors.” If it turned out that the error-free version of each DCP2 calculation came out to exactly the same value (the median cost-effectiveness figure for the distribution as a whole), that would be consistent with your subjective probabilities. Saying that the DCP2 is about as reliable as a repeated calculation with variation coming entirely from mistakes seems to me like a vote of extremely low confidence.
Hi all, regarding how far to generalize our discomfort with the DCP2 – I largely agree with what Alexander has said, but I also want to add another consideration.
As Alexander says, I think the errors are very worrying – they seem to reflect a process that has no systematic double-checks or reality-checks. However, the errors aren’t the only thing about the estimate that I find worth noting. I think it’s also worth noting the simplicity of the calculation.
Among other things, the calculation uses broad regional prevalence figures (and the final result is implicitly based on an equal-weighted average of regional prevalence, an incorrect way to take the average; Alexander didn’t mention this as an error), ignores externalities, ignores long-term considerations both positive and negative, ignores drug side-effects, and makes no apparent attempt to address the fact that its key figures (regarding delivery costs and effectiveness) are based on a small number of studies (i.e., there is no apparent attempt to address the question of how wastage and other execution issues would likely differ in larger-scale campaigns). There are no factors in the calculation that would make the estimate rise as evidence quality rises or fall as it falls. No sensitivity analysis is present.
We have conjectured that the estimates are this simple before, but now we know (for at least one estimate), and I don’t want to lose sight of this in the discussion about errors.
My impression is that the DCP2 estimate – far from trying to compensate for the complexity of deworming with sophistication of analysis – is basically a back-of-the-envelope figure that incorporates only a fraction of the information that we have at our disposal (if not in a way that can be effectively formalized) when investigating a specific charity. My impression is that other DCP2 estimates follow the same basic approach. Based on this alone, I would see little argument for giving them a substantial role in giving decisions. Adding in the high sensitivity to error that we’ve observed, and the apparent lack of a process for double-checks/reality-checks, makes me lean against giving them any role.
Given the amount of work it would take to further investigate this impression of the estimates, I’d rather focus our efforts on other work. Of course, if someone else came to us with evidence that other DCP2 estimates are more sophisticated, we’d be interested.
Will, when you say
are you saying that GiveWell went into this investigation intending to look for errors that overstated the cost-effectiveness of deworming, but not for errors that understated the cost-effectiveness of deworming? If so, what leads you to believe that?
This analysis is extremely useful – and I only wish it were more widely know. I wish all the problems underlying CE analysis were more widely known.
I believe you have only captured the tip of the iceberg of what’s wrong with CE analysis for health program interventions. A range of health system and other local context variables generate huge variation in the effectiveness AND cost of interventions.
The Miguel-Kremer paper that found school-based worming to be SO cost effective evaluated an intervention that was implemented in a unique way. The worming activities were implemented by an NGO which had been active in the district for a long time. NGO staff went to the schools to deliver the medicines to the children. And procured the drugs. And got the right amount to the right place at the right time. You would expect much lower effectiveness if a donor were supporting deworming in the context of a national program – with all the usual capacity problems coming in to play. In essence, many of the intervention studies underlying DCP2 figures are EFFICACY trials. They are not useful for predicting CE in real world implementation. Yes, there are studies of CE of deworming interventions implemented in the context of national programs, but since the syntheses of CE estimates do not separate efficacy trial CE figures from effectiveness, the results are not useful.
Furthermore, the studies do not take into account key health and country setting variables which we absolutely know to influence cost effectiveness. For example, again using Miguel-Kremer, the paper found very high cost effectiveness – but does not “deflate” for the very high school attendance in the Kenyan districts where the study was done 97-98%. The average in SSA is 63%. So, you would expect to reach fewer kids with the same program delivery strategy in an “average” African country setting. And indeed, when Leslie et al 2011 compared the side by side effectiveness of school based worming with community based worming in Niger (where school attendance is 38-40%) they found school based worming was much less cost effective than what Miguel-Kremer. NB: It is not possible to disentangle analytically the effect of two important differences in the Niger study – 1) the interventions were implemented in the context of a national program; and 2) much lower school attendance reduced the effectiveness of the school based program.
The upshot is: most CE numbers come from efficacy studies – or at least studies of interventions that are far from effectiveness studies. And, “average” results -really tell you little about what to expect in any particular place. Variations come from many of the factors you mentioned (particular types of worms; intensity of infection; population density) but also health (and other) system factors (school attendance; functionality of drug supply chains; strength of supervision of vector control program; functionality of local health facilities).
Keep up the good work!
Comments are closed.